Code
# install.packages("devtools")
devtools::install_github("cyplessen/metaMultiverse")Multiverse meta-analysis is an approach that addresses the analytical flexibility inherent in meta-analysis by exploring multiple reasonable analytical choices simultaneously. This package provides tools for conducting multiverse pairwise meta-analyses with standardized mean differences.
A multiverse analysis considers all reasonable analytical choices simultaneously, rather than committing to a single analytical path. This approach acknowledges the potential impact of researcher decisions on results and provides a more complete picture of possible outcomes.
The metaMultiverse package is designed to:
You can install the development version of metaMultiverse from GitHub:
# install.packages("devtools")
devtools::install_github("cyplessen/metaMultiverse")The typical workflow for a multiverse meta-analysis using this package consists of four main steps:
Prepare and validate your dataset to ensure it meets the requirements for multiverse analysis.
Define the analytical choices to explore, including “which” factors (data subsets) and “how” factors (meta-analytic methods).
Apply specifications to create a multiverse of meta-analyses, exploring all reasonable analytical paths.
Explore the results through specification curves and other visualizations to assess robustness.
This tutorial walks through each step with a practical example.
For this tutorial, we’ll use a dataset of digital depression interventions:
# Load required packages
library(metaMultiverse)
library(tidyr)
library(dplyr) # data wrangling
library(ggplot2) # data visualization
library(plotly) # interactive data visualization
library(kableExtra) # data formatting
# Load example dataset
data("data_digDep")
# Examine the structure
data_digDep %>%
kbl() %>%
kable_styling(
bootstrap_options = c("striped", "hover", "condensed", "responsive"),
full_width = FALSE,
position = "center"
) %>%
scroll_box(width = "100%", height = "300px")| study | es_id | yi | vi | sei | wf_1 | wf_2 | wf_3 | wf_4 | wf_5 | wf_6 | wf_7 | wf_8 | condition_arm1 | condition_arm2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Addington, 2019 | 1 | 0.2228951 | 0.1057582 | 0.3252049 | website | minimal to no support | adul | not-cbt-based | wl | cut | some concerns | post | other psy | wl |
| Addington, 2019 | 2 | 0.2734972 | 0.0981878 | 0.3133493 | website | minimal to no support | adul | not-cbt-based | other ctr | cut | some concerns | post | other psy | other ctr |
| Addington, 2019 | 3 | 0.3206356 | 0.1064319 | 0.3262391 | website | minimal to no support | adul | not-cbt-based | wl | cut | some concerns | post | other psy | wl |
| Addington, 2019 | 4 | 0.2134963 | 0.0978351 | 0.3127861 | website | minimal to no support | adul | not-cbt-based | other ctr | cut | some concerns | post | other psy | other ctr |
| Alavi, 2016 | 5 | 1.2933851 | 0.0521600 | 0.2283857 | website | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Alavi, 2016 | 6 | 1.3747302 | 0.0533464 | 0.2309685 | website | guided | adul | cbt-based | wl | cut | high risk | follow-up | cbt | wl |
| Andersson, 2005 | 7 | 0.9544523 | 0.0536431 | 0.2316098 | website | guided | adul | cbt-based | other ctr | cut | some concerns | post | cbt | other ctr |
| Andersson, 2005 | 8 | 0.7899661 | 0.0519243 | 0.2278690 | website | guided | adul | cbt-based | other ctr | cut | some concerns | post | cbt | other ctr |
| Araya, 2021a | 9 | 0.2965846 | 0.0069548 | 0.0833954 | mobile | guided | med | cbt-based | cau | cut | some concerns | post | bat | cau |
| Araya, 2021a | 10 | 0.1294997 | 0.0065934 | 0.0812000 | mobile | guided | med | cbt-based | cau | cut | some concerns | follow-up | bat | cau |
| Araya, 2021b | 11 | 0.4205505 | 0.0125420 | 0.1119913 | mobile | guided | med | cbt-based | cau | cut | some concerns | post | bat | cau |
| Araya, 2021b | 12 | 0.0863223 | 0.0122300 | 0.1105895 | mobile | guided | med | cbt-based | cau | cut | some concerns | follow-up | bat | cau |
| Arean, 2016 | 13 | 0.2665964 | 0.0290580 | 0.1704640 | mobile | minimal to no support | adul | not-cbt-based | other ctr | cut | some concerns | post | pst | other ctr |
| Arjadi, 2018 | 14 | 0.3886446 | 0.0130253 | 0.1141283 | website | guided | adul | cbt-based | other ctr | mood | some concerns | post | bat | other ctr |
| Baumeister, 2021 | 15 | 0.4458137 | 0.0196181 | 0.1400648 | website | guided | med | cbt-based | cau | mdd | some concerns | post | cbt | cau |
| Baumeister, 2021 | 16 | 0.2640861 | 0.0193073 | 0.1389506 | website | guided | med | cbt-based | cau | mdd | some concerns | follow-up | cbt | cau |
| Baumeister, 2021 | 17 | 0.2791261 | 0.0193269 | 0.1390214 | website | guided | med | cbt-based | cau | mdd | some concerns | post | cbt | cau |
| Baumeister, 2021 | 18 | 0.2318410 | 0.0192687 | 0.1388118 | website | guided | med | cbt-based | cau | mdd | some concerns | follow-up | cbt | cau |
| Baumeister, 2021 | 19 | 0.2138123 | 0.0192494 | 0.1387421 | website | guided | med | cbt-based | cau | mdd | some concerns | post | cbt | cau |
| Baumeister, 2021 | 20 | 0.1436914 | 0.0191889 | 0.1385242 | website | guided | med | cbt-based | cau | mdd | some concerns | follow-up | cbt | cau |
| Baumgartner, 2021 | 21 | 0.3175760 | 0.0089096 | 0.0943906 | website | guided | adul | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Baumgartner, 2021 | 22 | 0.2802350 | 0.0086312 | 0.0929041 | website | guided | adul | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Baumgartner, 2021 | 23 | 0.3846998 | 0.0089616 | 0.0946655 | website | guided | adul | cbt-based | wl | cut | some concerns | follow-up | cbt | wl |
| Baumgartner, 2021 | 24 | 0.3772827 | 0.0086996 | 0.0932715 | website | guided | adul | cbt-based | wl | cut | some concerns | follow-up | cbt | wl |
| Beevers, 2017 | 25 | 0.8144986 | 0.0153835 | 0.1240303 | website | automated encouragement | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Beevers, 2017 | 26 | 0.7632797 | 0.0152756 | 0.1235946 | website | automated encouragement | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Bendig, 2021 | 27 | 0.3154515 | 0.1195906 | 0.3458187 | website | guided | med | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Berger, 2011 | 28 | 0.6540990 | 0.0827882 | 0.2877294 | website | minimal to no support | adul | cbt-based | wl | mood | some concerns | post | cbt | wl |
| Berger, 2011 | 29 | 0.1442425 | 0.0786719 | 0.2804852 | website | minimal to no support | adul | cbt-based | wl | mood | some concerns | follow-up | cbt | wl |
| Berger, 2011 | 30 | 1.1252747 | 0.0912667 | 0.3021037 | website | guided | adul | cbt-based | wl | mood | some concerns | post | cbt | wl |
| Berger, 2011 | 31 | 0.4470027 | 0.0804822 | 0.2836938 | website | guided | adul | cbt-based | wl | mood | some concerns | follow-up | cbt | wl |
| Birney, 2016 | 32 | 0.2900000 | 0.0144000 | 0.1200000 | mobile | automated encouragement | other group | cbt-based | other ctr | cut | some concerns | post | cbt | other ctr |
| Blanco, 2023 | 33 | 1.4800146 | 0.0884493 | 0.2974042 | mobile | guided | adul | cbt-based | other ctr | cut | low risk | post | cbt | other ctr |
| Blanco, 2023 | 34 | 0.6443551 | 0.0714296 | 0.2672631 | mobile | automated encouragement | adul | cbt-based | other ctr | cut | low risk | post | cbt | other ctr |
| Boele, 2018 | 35 | 0.6102799 | 0.0856631 | 0.2926827 | website | guided | med | not-cbt-based | wl | cut | high risk | post | pst | wl |
| Boeschoten, 2017 | 36 | 0.0777749 | 0.0263541 | 0.1623395 | website | guided | med | not-cbt-based | wl | cut | low risk | post | pst | wl |
| Braun, 2021 | 37 | 0.2806605 | 0.0118815 | 0.1090021 | website | guided | other group | not-cbt-based | other ctr | cut | some concerns | post | other psy | other ctr |
| Buntrock, 2015 | 38 | 0.6583983 | 0.0103883 | 0.1019230 | website | guided | adul | not-cbt-based | cau | cut | some concerns | post | other psy | cau |
| Buntrock, 2015 | 39 | 0.2823902 | 0.0099510 | 0.0997548 | website | guided | adul | not-cbt-based | cau | cut | some concerns | follow-up | other psy | cau |
| Carlbring, 2013 | 40 | 0.8545814 | 0.0546538 | 0.2337815 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | 3rd | wl |
| Carlbring, 2013 | 41 | 0.6330426 | 0.0525537 | 0.2292459 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | 3rd | wl |
| Choi, 2012 | 42 | 0.8934932 | 0.0700034 | 0.2645816 | website | guided | other group | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Choi, 2012 | 43 | 0.4805137 | 0.0653867 | 0.2557081 | website | guided | other group | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Clarke, 2019 | 44 | -0.0786665 | 0.0084663 | 0.0920124 | website | automated encouragement | med | cbt-based | other ctr | cut | some concerns | post | cbt | other ctr |
| Cooper, 2011 | 45 | 0.8280493 | 0.2121586 | 0.4606068 | both | minimal to no support | med | cbt-based | cau | cut | high risk | post | cbt | cau |
| Cuijpers, 2022a | 46 | -0.1243773 | 0.1856208 | 0.4308373 | mobile | guided | other group | cbt-based | other ctr | cut | low risk | post | bat | other ctr |
| Cuijpers, 2022b | 47 | 0.7325632 | 0.0186483 | 0.1365588 | mobile | guided | adul | cbt-based | other ctr | cut | low risk | post | bat | other ctr |
| Dahne, 2019a | 48 | 0.5280632 | 0.1523608 | 0.3903342 | mobile | automated encouragement | other group | cbt-based | cau | cut | high risk | post | bat | cau |
| Dahne, 2019a | 49 | 0.4517191 | 0.2169961 | 0.4658284 | mobile | minimal to no support | other group | cbt-based | cau | cut | high risk | post | cbt | cau |
| Dahne, 2019b | 50 | 0.3174645 | 0.1582794 | 0.3978435 | mobile | minimal to no support | adul | cbt-based | cau | cut | high risk | post | bat | cau |
| Dahne, 2019b | 51 | 0.4274193 | 0.1702645 | 0.4126312 | mobile | minimal to no support | adul | cbt-based | cau | cut | high risk | post | cbt | cau |
| Danaher, 2022 | 52 | 0.3300421 | 0.0212304 | 0.1457066 | website | automated encouragement | ppd | cbt-based | cau | cut | some concerns | post | cbt | cau |
| Deady, 2016 | 53 | 0.6229982 | 0.0753591 | 0.2745160 | website | minimal to no support | other group | cbt-based | other ctr | cut | some concerns | post | cbt | other ctr |
| Deady, 2016 | 54 | 0.2936700 | 0.1052888 | 0.3244824 | website | minimal to no support | other group | cbt-based | other ctr | cut | some concerns | follow-up | cbt | other ctr |
| deGraaf, 2009 | 55 | 0.1451012 | 0.0197610 | 0.1405738 | both | minimal to no support | adul | cbt-based | cau | cut | low risk | post | cbt | cau |
| deGraaf, 2009 | 56 | 0.0392583 | 0.0197126 | 0.1404014 | both | minimal to no support | adul | cbt-based | cau | cut | low risk | post | cbt | NA |
| deGraaf, 2009 | 57 | 0.0976294 | 0.0197324 | 0.1404720 | both | minimal to no support | adul | cbt-based | cau | cut | low risk | follow-up | cbt | cau |
| deGraaf, 2009 | 58 | 0.1162436 | 0.0197423 | 0.1405072 | both | human encouragement | adul | cbt-based | cau | cut | low risk | follow-up | cbt | cau |
| Ebert, 2014 | 59 | 0.6905201 | 0.0282723 | 0.1681437 | website | guided | other group | not-cbt-based | wl | cut | some concerns | post | pst | wl |
| Ebert, 2014 | 60 | 0.3806003 | 0.0271545 | 0.1647861 | website | guided | other group | not-cbt-based | wl | cut | some concerns | follow-up | pst | wl |
| Ebert, 2018 | 61 | 0.3997118 | 0.0200024 | 0.1414297 | website | human encouragement | adul | not-cbt-based | wl | cut | some concerns | post | other psy | wl |
| Ebert, 2018 | 62 | 0.3459149 | 0.0199033 | 0.1410791 | website | human encouragement | adul | not-cbt-based | wl | cut | some concerns | post | other psy | wl |
| Ebert, 2018 | 63 | 0.8383371 | 0.0213433 | 0.1460934 | website | human encouragement | adul | not-cbt-based | wl | cut | some concerns | post | other psy | wl |
| Flygare, 2020 | 64 | 0.2306238 | 0.0423944 | 0.2058991 | website | guided | adul | cbt-based | other ctr | cut | some concerns | post | cbt | other ctr |
| Flygare, 2020 | 65 | 0.2635909 | 0.0424816 | 0.2061107 | website | guided | adul | cbt-based | other ctr | cut | some concerns | follow-up | cbt | other ctr |
| Flygare, 2020 | 66 | 0.1293489 | 0.0421994 | 0.2054250 | website | guided | adul | cbt-based | other ctr | cut | some concerns | follow-up | cbt | other ctr |
| Flygare, 2020 | 67 | 0.2279166 | 0.0423878 | 0.2058830 | website | guided | adul | cbt-based | other ctr | cut | some concerns | post | cbt | other ctr |
| Flygare, 2020 | 68 | 0.1761181 | 0.0422759 | 0.2056109 | website | guided | adul | cbt-based | other ctr | cut | some concerns | follow-up | cbt | other ctr |
| Flygare, 2020 | 69 | 0.0776440 | 0.0421422 | 0.2052856 | website | guided | adul | cbt-based | other ctr | cut | some concerns | follow-up | cbt | other ctr |
| Fonseca, 2020 | 70 | -0.0093911 | 0.0206210 | 0.1436001 | website | minimal to no support | ppd | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Forand, 2018 | 71 | 1.5943651 | 0.0648136 | 0.2545851 | both | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Forand, 2018 | 72 | 1.5537020 | 0.0763895 | 0.2763865 | both | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Forsell, 2017 | 73 | 1.2109857 | 0.1227672 | 0.3503816 | website | guided | ppd | cbt-based | cau | mdd | some concerns | post | cbt | cau |
| Forsell, 2017 | 74 | 0.5188209 | 0.1153655 | 0.3396550 | website | guided | ppd | cbt-based | cau | mdd | some concerns | post | cbt | cau |
| Gaudiano, 2020 | 75 | -0.5802882 | 0.1046322 | 0.3234690 | website | minimal to no support | adul | cbt-based | other ctr | mdd | high risk | post | 3rd | other ctr |
| Geraedts, 2014 | 76 | 0.2521615 | 0.0174549 | 0.1321169 | website | guided | other group | cbt-based | cau | cut | some concerns | post | cbt | cau |
| Ghosh, 2021 | 77 | 0.9002485 | 0.1528561 | 0.3909681 | website | minimal to no support | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Ghosh, 2021 | 78 | 0.3480962 | 0.0530287 | 0.2302796 | website | automated encouragement | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Gilbody, 2015 | 79 | -0.0733015 | 0.0112938 | 0.1062721 | website | human encouragement | adul | cbt-based | cau | cut | high risk | post | cbt | cau |
| Gilbody, 2015 | 80 | -0.0876416 | 0.0105206 | 0.1025701 | website | human encouragement | adul | cbt-based | cau | cut | high risk | post | cbt | cau |
| Gilbody, 2015 | 81 | 0.1048457 | 0.0163444 | 0.1278454 | website | human encouragement | adul | cbt-based | cau | cut | high risk | follow-up | cbt | cau |
| Gilbody, 2015 | 82 | 0.2295962 | 0.0163676 | 0.1279358 | website | human encouragement | adul | cbt-based | cau | cut | high risk | follow-up | cbt | cau |
| Gilbody, 2015 | 83 | -0.0766299 | 0.0168449 | 0.1297880 | website | human encouragement | adul | cbt-based | cau | cut | high risk | follow-up | cbt | cau |
| Gilbody, 2015 | 84 | 0.1005381 | 0.0165382 | 0.1286008 | website | human encouragement | adul | cbt-based | cau | cut | high risk | follow-up | cbt | cau |
| Gili, 2020 | 85 | 0.2300000 | 0.0361000 | 0.1900000 | website | automated encouragement | adul | not-cbt-based | cau | mood | some concerns | post | other psy | cau |
| Gili, 2020 | 86 | 0.3100000 | 0.0361000 | 0.1900000 | website | automated encouragement | adul | not-cbt-based | cau | mood | some concerns | follow-up | other psy | cau |
| Gili, 2020 | 87 | 0.0100000 | 0.5184000 | 0.7200000 | website | automated encouragement | adul | not-cbt-based | cau | mood | some concerns | follow-up | other psy | cau |
| Glozier, 2013 | 88 | 0.1618894 | 0.0083629 | 0.0914488 | website | minimal to no support | med | cbt-based | other ctr | cut | low risk | post | cbt | other ctr |
| Gräfe, 2020 | 89 | 0.2100000 | 0.0009000 | 0.0300000 | website | minimal to no support | adul | cbt-based | cau | cut | high risk | post | cbt | cau |
| Gräfe, 2020 | 90 | 0.2300000 | 0.0016000 | 0.0400000 | website | minimal to no support | adul | cbt-based | cau | cut | high risk | follow-up | cbt | cau |
| Gräfe, 2020 | 91 | 0.1500000 | 0.0016000 | 0.0400000 | website | minimal to no support | adul | cbt-based | cau | cut | high risk | follow-up | cbt | cau |
| Guo, 2020 | 92 | 0.6238601 | 0.0139853 | 0.1182594 | mobile | human encouragement | med | not-cbt-based | wl | cut | low risk | post | other psy | wl |
| Guo, 2020 | 93 | 0.6012903 | 0.0139390 | 0.1180634 | mobile | human encouragement | med | not-cbt-based | wl | cut | low risk | follow-up | other psy | wl |
| Guo, 2020 | 94 | 0.5122734 | 0.0137729 | 0.1173581 | mobile | human encouragement | med | not-cbt-based | wl | cut | low risk | follow-up | other psy | wl |
| Guo, 2020 | 95 | 0.4749678 | 0.0137112 | 0.1170949 | mobile | human encouragement | med | not-cbt-based | wl | cut | low risk | post | other psy | wl |
| Guo, 2020 | 96 | 0.4979126 | 0.0137486 | 0.1172545 | mobile | human encouragement | med | not-cbt-based | wl | cut | low risk | follow-up | other psy | wl |
| Guo, 2020 | 97 | 0.3418540 | 0.0135291 | 0.1163146 | mobile | human encouragement | med | not-cbt-based | wl | cut | low risk | follow-up | other psy | wl |
| Gupta, 2020 | 98 | -0.2700000 | 0.0100000 | 0.1000000 | website | human encouragement | med | cbt-based | cau | cut | high risk | post | cbt | cau |
| Gupta, 2020 | 99 | -0.2800000 | 0.0121000 | 0.1100000 | website | human encouragement | med | cbt-based | cau | cut | high risk | follow-up | cbt | cau |
| Gupta, 2020 | 100 | -0.0600000 | 0.0064000 | 0.0800000 | website | human encouragement | med | cbt-based | cau | cut | high risk | post | cbt | cau |
| Gupta, 2020 | 101 | -0.1700000 | 0.0064000 | 0.0800000 | website | human encouragement | med | cbt-based | cau | cut | high risk | follow-up | cbt | cau |
| Hallgren, 2015 | 102 | 0.3206799 | 0.0082764 | 0.0909747 | website | guided | adul | cbt-based | cau | cut | high risk | post | cbt | cau |
| Harrer, 2021 | 103 | 0.3644587 | 0.0203346 | 0.1425995 | website | automated encouragement | young | not-cbt-based | other ctr | cut | some concerns | post | other psy | other ctr |
| Heim, 2021 | 104 | 0.7019164 | 0.0956716 | 0.3093082 | mobile | guided | adul | cbt-based | other ctr | cut | high risk | post | bat | other ctr |
| Hobfoll, 2016 | 105 | 0.2191702 | 0.0155027 | 0.1245096 | website | human encouragement | other group | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Hoifodt, 2013 | 106 | 0.9235597 | 0.0418315 | 0.2045275 | website | guided | adul | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Hoifodt, 2013 | 107 | 0.5233640 | 0.0390602 | 0.1976365 | website | guided | adul | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Hoifodt, 2013 | 108 | -0.3173972 | 0.0382314 | 0.1955286 | website | guided | adul | cbt-based | wl | cut | some concerns | follow-up | cbt | wl |
| Hoifodt, 2013 | 109 | -0.1489189 | 0.0378554 | 0.1945647 | website | guided | adul | cbt-based | wl | cut | some concerns | follow-up | cbt | wl |
| Hur, 2018 | 110 | 0.6616861 | 0.1244010 | 0.3527053 | mobile | minimal to no support | adul | cbt-based | other ctr | mood | high risk | post | cbt | other ctr |
| Jannati, 2020 | 111 | 2.9462647 | 0.1080446 | 0.3287013 | mobile | minimal to no support | ppd | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Jelinek, 2020 | 112 | 0.2786943 | 0.0722330 | 0.2687621 | website | minimal to no support | adul | cbt-based | other ctr | cut | some concerns | post | bat | other ctr |
| Jelinek, 2020 | 113 | 0.0381744 | 0.0657450 | 0.2564079 | website | minimal to no support | adul | cbt-based | cau | cut | some concerns | post | bat | cau |
| Johansson, 2012a | 114 | 1.1111492 | 0.0503019 | 0.2242808 | website | guided | adul | not-cbt-based | wl | mdd | some concerns | post | dyn | wl |
| Johansson, 2012a | 115 | 0.9368068 | 0.0483286 | 0.2198376 | website | guided | adul | not-cbt-based | wl | mdd | some concerns | post | dyn | wl |
| Johansson, 2012a | 116 | 0.8492545 | 0.0474643 | 0.2178631 | website | guided | adul | not-cbt-based | wl | mdd | some concerns | post | dyn | wl |
| Johansson, 2012b | 117 | 0.8261220 | 0.0580639 | 0.2409646 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Johansson, 2012b | 118 | 0.7903957 | 0.0576709 | 0.2401476 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Johansson, 2012b | 119 | 0.5691130 | 0.0573190 | 0.2394139 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Johansson, 2012b | 120 | 0.5590523 | 0.0572396 | 0.2392480 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Johansson, 2019a | 121 | 1.5859228 | 0.0980525 | 0.3131334 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Johansson, 2019a | 122 | 1.5378623 | 0.0966212 | 0.3108395 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Johansson, 2019b | 123 | 0.4356529 | 0.0284438 | 0.1686529 | website | guided | med | cbt-based | wl | cut | high risk | post | cbt | wl |
| Johansson, 2019b | 124 | 0.6722977 | 0.0293639 | 0.1713590 | website | guided | med | cbt-based | wl | cut | high risk | post | cbt | wl |
| Kivi, 2014 | 125 | 0.1570575 | 0.0658588 | 0.2566296 | website | guided | adul | cbt-based | cau | mood | high risk | post | cbt | cau |
| Kramer, 2021 | 126 | 0.5181935 | 0.0304165 | 0.1744033 | website | guided | adul | not-cbt-based | wl | cut | some concerns | post | other psy | wl |
| Kramer, 2021 | 127 | 0.5192139 | 0.0304205 | 0.1744146 | website | guided | adul | not-cbt-based | wl | cut | some concerns | post | other psy | wl |
| Krämer, 2022 | 128 | 1.4360547 | 0.0208163 | 0.1442784 | website | minimal to no support | adul | not-cbt-based | wl | mood | high risk | post | other psy | wl |
| Krämer, 2022 | 129 | 2.3409554 | 0.0276935 | 0.1664136 | website | minimal to no support | adul | not-cbt-based | wl | mood | high risk | post | other psy | wl |
| Krämer, 2022 | 130 | 0.9217970 | 0.0183764 | 0.1355597 | website | minimal to no support | adul | not-cbt-based | wl | mood | high risk | post | other psy | wl |
| Krämer, 2022 | 131 | 2.8290555 | 0.0953401 | 0.3087720 | website | minimal to no support | adul | not-cbt-based | wl | mood | high risk | follow-up | other psy | wl |
| Krämer, 2022 | 132 | 1.8889909 | 0.0705534 | 0.2656189 | website | minimal to no support | adul | not-cbt-based | wl | mood | high risk | follow-up | other psy | wl |
| Krämer, 2022 | 133 | 3.2916609 | 0.7911097 | 0.8894435 | website | minimal to no support | adul | not-cbt-based | wl | mood | high risk | follow-up | other psy | wl |
| Krämer, 2022 | 134 | 1.5792607 | 0.0216209 | 0.1470404 | website | guided | adul | cbt-based | wl | mood | high risk | post | cbt | wl |
| Krämer, 2022 | 135 | 2.6100029 | 0.0302747 | 0.1739962 | website | guided | adul | cbt-based | wl | mood | high risk | post | cbt | wl |
| Krämer, 2022 | 136 | 1.2669374 | 0.0198394 | 0.1408523 | website | guided | adul | cbt-based | wl | mood | high risk | post | cbt | wl |
| Krämer, 2022 | 137 | 2.4263728 | 0.0822896 | 0.2868616 | website | guided | adul | cbt-based | wl | mood | high risk | follow-up | cbt | wl |
| Krämer, 2022 | 138 | 2.0904062 | 0.0734957 | 0.2711009 | website | guided | adul | cbt-based | wl | mood | high risk | follow-up | cbt | wl |
| Krämer, 2022 | 139 | 2.1306335 | 0.3031316 | 0.5505739 | website | guided | adul | cbt-based | wl | mood | high risk | follow-up | cbt | wl |
| Lambert, 2018 | 140 | 0.9166389 | 0.0886726 | 0.2977794 | website | human encouragement | adul | cbt-based | wl | cut | high risk | post | bat | wl |
| Lappalainen, 2015 | 141 | 0.6247423 | 0.1109136 | 0.3330369 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | 3rd | wl |
| Lobner, 2018 | 142 | 0.0806035 | 0.0074494 | 0.0863101 | website | minimal to no support | adul | cbt-based | cau | mood | some concerns | post | cbt | cau |
| Lobner, 2018 | 143 | 0.1726839 | 0.0085106 | 0.0922529 | website | minimal to no support | adul | cbt-based | cau | mood | some concerns | follow-up | cbt | cau |
| Lobner, 2018 | 144 | 0.0000000 | 0.0072999 | 0.0854397 | website | minimal to no support | adul | cbt-based | cau | mood | some concerns | post | cbt | cau |
| Lobner, 2018 | 145 | 0.1350230 | 0.0083799 | 0.0915416 | website | minimal to no support | adul | cbt-based | cau | mood | some concerns | follow-up | cbt | cau |
| Lokman, 2017 | 146 | 0.3867331 | 0.0123865 | 0.1112947 | website | automated encouragement | adul | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Lu, 2021 | 147 | 0.6648098 | 0.1217449 | 0.3489196 | website | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Lukas, 2021 | 148 | 1.2141225 | 0.0617941 | 0.2485841 | mobile | automated encouragement | adul | not-cbt-based | wl | cut | high risk | post | other psy | wl |
| Lundgren, 2016 | 149 | 0.2652767 | 0.0807264 | 0.2841238 | website | guided | med | cbt-based | wl | cut | some concerns | post | cbt | wl |
| MacKinnon, 2022 | 150 | 0.5146510 | 0.1721647 | 0.4149274 | mobile | guided | other group | not-cbt-based | cau | cut | high risk | post | other psy | cau |
| MacLean, 2020 | 151 | 0.1704545 | 0.0431745 | 0.2077849 | website | guided | adul | not-cbt-based | other ctr | cut | low risk | post | other psy | other ctr |
| Meyer, 2015 | 152 | 0.5668467 | 0.0255801 | 0.1599378 | website | automated encouragement | adul | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Meyer, 2015 | 153 | 0.3316252 | 0.0249257 | 0.1578789 | website | automated encouragement | adul | cbt-based | wl | cut | some concerns | follow-up | cbt | wl |
| Meyer, 2019 | 154 | 0.5369133 | 0.0207262 | 0.1439659 | website | minimal to no support | med | cbt-based | wl | mood | some concerns | post | cbt | wl |
| Meyer, 2019 | 155 | 0.5076201 | 0.0206491 | 0.1436980 | website | minimal to no support | med | cbt-based | wl | mood | some concerns | post | cbt | wl |
| Milgrom, 2016 | 156 | 0.8285112 | 0.1013575 | 0.3183669 | website | human encouragement | ppd | cbt-based | cau | mood | low risk | post | cbt | cau |
| Milgrom, 2021 | 157 | 0.7468071 | 0.0556522 | 0.2359071 | website | human encouragement | ppd | cbt-based | cau | mood | low risk | post | cbt | cau |
| Moeini, 2019 | 158 | 0.1944934 | 0.0313995 | 0.1771992 | website | guided | young | cbt-based | cau | cut | some concerns | post | cbt | cau |
| Moeini, 2019 | 159 | 0.2589615 | 0.0315151 | 0.1775250 | website | guided | young | cbt-based | cau | cut | some concerns | follow-up | cbt | cau |
| Mohr, 2013 | 160 | 0.3964969 | 0.0600572 | 0.2450658 | website | minimal to no support | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Mohr, 2013 | 161 | 0.5174917 | 0.0617604 | 0.2485164 | website | human encouragement | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Montero-Marin, 2016 | 162 | 0.1208954 | 0.0325289 | 0.1803577 | website | automated encouragement | adul | cbt-based | cau | mdd | low risk | post | cbt | cau |
| Montero-Marin, 2016 | 163 | 0.3411935 | 0.0329445 | 0.1815061 | website | automated encouragement | adul | cbt-based | cau | mdd | low risk | follow-up | cbt | cau |
| Montero-Marin, 2016 | 164 | 0.4751333 | 0.0333908 | 0.1827316 | website | automated encouragement | adul | cbt-based | cau | mdd | low risk | follow-up | cbt | cau |
| Montero-Marin, 2016 | 165 | 0.0773538 | 0.0305735 | 0.1748527 | website | guided | adul | cbt-based | cau | mdd | low risk | post | cbt | cau |
| Montero-Marin, 2016 | 166 | 0.3820679 | 0.0311141 | 0.1763918 | website | guided | adul | cbt-based | cau | mdd | low risk | follow-up | cbt | cau |
| Montero-Marin, 2016 | 167 | 0.4832553 | 0.0314522 | 0.1773477 | website | guided | adul | cbt-based | cau | mdd | low risk | follow-up | cbt | cau |
| Morgan, 2012 | 168 | 0.1644991 | 0.0070600 | 0.0840238 | website | minimal to no support | adul | not-cbt-based | other ctr | cut | low risk | post | other psy | other ctr |
| Nadort, 2022 | 169 | 0.0616223 | 0.0332402 | 0.1823188 | website | guided | med | not-cbt-based | cau | cut | some concerns | post | pst | cau |
| Newby, 2017 | 170 | 0.7763902 | 0.0579911 | 0.2408134 | website | guided | med | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Nobis, 2015 | 171 | 0.8914534 | 0.0171873 | 0.1311003 | website | guided | med | not-cbt-based | other ctr | cut | some concerns | post | other psy | other ctr |
| Nobis, 2015 | 172 | 0.8191106 | 0.0169442 | 0.1301698 | website | guided | med | not-cbt-based | other ctr | cut | some concerns | post | other psy | other ctr |
| Nyström, 2017 | 173 | 0.9727684 | 0.0388740 | 0.1971650 | website | guided | adul | cbt-based | wl | mood | low risk | post | bat | wl |
| Nyström, 2017 | 174 | 0.6490930 | 0.0413728 | 0.2034031 | website | guided | adul | cbt-based | wl | mood | low risk | post | bat | wl |
| O' Moore, 2018 | 175 | 1.0077723 | 0.0702549 | 0.2650565 | website | minimal to no support | med | cbt-based | cau | mdd | some concerns | post | cbt | cau |
| O' Moore, 2018 | 176 | 0.9015138 | 0.0687512 | 0.2622045 | website | minimal to no support | med | cbt-based | cau | mdd | some concerns | follow-up | cbt | cau |
| O'Mahen, 2013a | 181 | 0.6472454 | 0.0594544 | 0.2438328 | website | guided | ppd | cbt-based | cau | mdd | some concerns | post | bat | cau |
| O'Mahen, 2013a | 182 | 0.4802299 | 0.0699795 | 0.2645363 | website | guided | ppd | cbt-based | cau | mdd | some concerns | follow-up | bat | cau |
| O'Mahen, 2013b | 183 | 0.5469195 | 0.0121357 | 0.1101620 | website | automated encouragement | ppd | cbt-based | cau | cut | some concerns | post | bat | cau |
| Oehler, 2020 | 177 | 0.2328341 | 0.0152626 | 0.1235419 | website | guided | adul | cbt-based | other ctr | mood | some concerns | post | cbt | other ctr |
| Oehler, 2020 | 178 | 0.1347449 | 0.0153056 | 0.1237157 | website | guided | adul | cbt-based | other ctr | mood | some concerns | post | cbt | other ctr |
| Oehler, 2020 | 179 | 0.1230026 | 0.0170787 | 0.1306854 | website | guided | adul | cbt-based | other ctr | mood | some concerns | follow-up | cbt | other ctr |
| Oehler, 2020 | 180 | 0.1113171 | 0.0170728 | 0.1306630 | website | guided | adul | cbt-based | other ctr | mood | some concerns | follow-up | cbt | other ctr |
| Perini, 2009 | 184 | 0.8469905 | 0.1008508 | 0.3175702 | website | guided | adul | cbt-based | wl | mdd | high risk | post | cbt | wl |
| Perini, 2009 | 185 | 0.6138469 | 0.1003006 | 0.3167026 | website | guided | adul | cbt-based | wl | mdd | high risk | post | cbt | wl |
| Pfeiffer, 2020 | 186 | 0.1364048 | 0.0171114 | 0.1308107 | website | guided | other group | cbt-based | cau | cut | some concerns | post | cbt | cau |
| Pfeiffer, 2020 | 187 | 0.1262654 | 0.0184122 | 0.1356916 | website | guided | other group | cbt-based | cau | cut | some concerns | follow-up | cbt | cau |
| Phillips, 2014 | 188 | 0.0494895 | 0.0117830 | 0.1085495 | website | human encouragement | other group | cbt-based | other ctr | cut | some concerns | post | cbt | other ctr |
| Pots, 2016 | 189 | 0.5581645 | 0.0246195 | 0.1569059 | website | guided | adul | cbt-based | wl | cut | high risk | post | 3rd | wl |
| Pots, 2016 | 190 | 0.3187602 | 0.0239927 | 0.1548958 | website | guided | adul | cbt-based | wl | cut | high risk | follow-up | 3rd | wl |
| Pugh, 2016 | 191 | 1.0539618 | 0.1147047 | 0.3386808 | website | guided | ppd | cbt-based | wl | cut | high risk | post | cbt | wl |
| Pugh, 2016 | 192 | 0.8770698 | 0.1102600 | 0.3320543 | website | guided | ppd | cbt-based | wl | cut | high risk | post | cbt | wl |
| Raevuori, 2021 | 193 | -0.2581432 | 0.0325385 | 0.1803843 | mobile | guided | young | not-cbt-based | cau | mdd | some concerns | post | other psy | cau |
| Raevuori, 2021 | 194 | 0.1005334 | 0.0323077 | 0.1797435 | mobile | guided | young | not-cbt-based | cau | mdd | some concerns | follow-up | other psy | cau |
| Reins, 2019 | 195 | 0.3163818 | 0.0309227 | 0.1758484 | website | guided | adul | not-cbt-based | other ctr | mdd | some concerns | post | other psy | other ctr |
| Reins, 2019 | 196 | 0.3015156 | 0.0308872 | 0.1757475 | website | guided | adul | not-cbt-based | other ctr | mdd | some concerns | post | other psy | other ctr |
| Reins, 2019 | 197 | 0.2836912 | 0.0308469 | 0.1756329 | website | guided | adul | not-cbt-based | other ctr | mdd | some concerns | post | other psy | other ctr |
| Richards, 2015 | 198 | 0.6457757 | 0.0224044 | 0.1496808 | website | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Richki, 2015a | 199 | 1.1394887 | 0.1586061 | 0.3982539 | website | guided | young | not-cbt-based | wl | mdd | low risk | post | other psy | wl |
| Richki, 2015b | 200 | 0.8174530 | 0.1405297 | 0.3748729 | website | guided | young | not-cbt-based | wl | mdd | some concerns | post | other psy | wl |
| Ritvo, 2021 | 201 | 0.4429332 | 0.0911912 | 0.3019789 | website | guided | young | cbt-based | cau | mdd | high risk | post | 3rd | cau |
| Ritvo, 2021 | 202 | 0.7014920 | 0.0945975 | 0.3075671 | website | guided | young | cbt-based | cau | mdd | high risk | post | 3rd | cau |
| Ritvo, 2021 | 203 | 0.6947830 | 0.0944897 | 0.3073917 | website | guided | young | cbt-based | cau | mdd | high risk | post | 3rd | cau |
| Roepke, 2015 | 204 | 0.3180025 | 0.0787064 | 0.2805466 | website | minimal to no support | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Roepke, 2015 | 205 | 0.7772986 | 0.0890935 | 0.2984853 | website | minimal to no support | adul | cbt-based | wl | cut | high risk | post | 3rd | wl |
| Rosso, 2017 | 206 | 0.7856270 | 0.0561165 | 0.2368893 | website | human encouragement | adul | cbt-based | other ctr | mdd | low risk | post | cbt | other ctr |
| Rosso, 2017 | 207 | 0.7811842 | 0.0560704 | 0.2367919 | website | human encouragement | adul | cbt-based | other ctr | mdd | low risk | post | cbt | other ctr |
| Ruehlman, 2021 | 208 | 0.7721817 | 0.0832911 | 0.2886020 | website | minimal to no support | young | cbt-based | wl | cut | high risk | post | cbt | wl |
| Ruwaard, 2009 | 209 | 0.8473307 | 0.0901782 | 0.3002968 | website | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Ruwaard, 2009 | 210 | 0.8309269 | 0.0899157 | 0.2998595 | website | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Ruwaard, 2009 | 211 | 0.5196580 | 0.0859078 | 0.2931004 | website | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Salamanca-Sanabria, 2020 | 212 | 0.8979870 | 0.0716260 | 0.2676304 | website | guided | young | cbt-based | wl | cut | high risk | post | cbt | wl |
| Sander, 2020 | 215 | 0.4228443 | 0.0138653 | 0.1177511 | website | guided | med | cbt-based | cau | cut | some concerns | post | cbt | cau |
| Sander, 2020 | 216 | 0.2753403 | 0.0136899 | 0.1170038 | website | guided | med | cbt-based | cau | cut | some concerns | follow-up | cbt | cau |
| Sander, 2020 | 217 | 0.4021581 | 0.0138363 | 0.1176276 | website | guided | med | cbt-based | cau | cut | some concerns | follow-up | cbt | cau |
| Sander, 2020 | 218 | 0.3284546 | 0.0137445 | 0.1172370 | website | guided | med | cbt-based | cau | cut | some concerns | follow-up | cbt | cau |
| Sander, 2020 | 219 | 0.3429331 | 0.0137611 | 0.1173076 | website | guided | med | cbt-based | cau | cut | some concerns | follow-up | cbt | cau |
| Sawyer, 2019 | 262 | 0.2080483 | 0.0362601 | 0.1904208 | mobile | guided | ppd | cbt-based | cau | cut | low risk | post | cbt | cau |
| Sawyer, 2019 | 263 | -0.2496580 | 0.0363471 | 0.1906490 | mobile | guided | ppd | cbt-based | cau | cut | low risk | follow-up | cbt | cau |
| Schleider, 2022 | 213 | 0.1969190 | 0.0024644 | 0.0496428 | website | minimal to no support | young | not-cbt-based | other ctr | cut | some concerns | post | other psy | other ctr |
| Schleider, 2022 | 214 | 0.2196713 | 0.0024553 | 0.0495505 | website | minimal to no support | young | cbt-based | other ctr | cut | some concerns | post | bat | other ctr |
| Schlicker, 2020 | 220 | 0.4255328 | 0.0539937 | 0.2323654 | website | guided | med | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Schlicker, 2020 | 221 | 0.2972396 | 0.0533710 | 0.2310217 | website | guided | med | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Schlicker, 2020 | 222 | 0.2466567 | 0.0531863 | 0.2306216 | website | guided | med | cbt-based | wl | mdd | some concerns | follow-up | cbt | wl |
| Schlicker, 2020 | 223 | 0.3714769 | 0.0537044 | 0.2317421 | website | guided | med | cbt-based | wl | mdd | some concerns | follow-up | cbt | wl |
| Schure, 2019 | 224 | 0.5255538 | 0.0121021 | 0.1100096 | website | automated encouragement | adul | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Seo, 2022 | 225 | 0.4222655 | 0.0560524 | 0.2367539 | mobile | human encouragement | ppd | cbt-based | other ctr | cut | high risk | post | cbt | other ctr |
| Sheeber, 2012 | 226 | 0.8402299 | 0.0632160 | 0.2514279 | website | guided | ppd | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Sheeber, 2017 | 227 | 0.3133597 | 0.0152241 | 0.1233859 | website | guided | ppd | cbt-based | other ctr | cut | some concerns | post | cbt | other ctr |
| Sheeber, 2017 | 228 | 0.3083201 | 0.0152182 | 0.1233619 | website | guided | ppd | cbt-based | other ctr | cut | some concerns | post | cbt | other ctr |
| Smith, 2017 | 229 | 0.8550238 | 0.0557363 | 0.2360854 | website | human encouragement | adul | cbt-based | wl | mdd | high risk | post | cbt | wl |
| Spek, 2007 | 230 | 0.2667666 | 0.0199814 | 0.1413556 | website | minimal to no support | adul | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Stiles-Shields, 2019 | 231 | 0.9311597 | 0.2236313 | 0.4728967 | mobile | human encouragement | adul | cbt-based | wl | cut | high risk | post | bat | wl |
| Stiles-Shields, 2019 | 232 | 1.5085550 | 0.3171542 | 0.5631644 | mobile | human encouragement | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Stuart, 2022 | 233 | 0.5563872 | 0.0239751 | 0.1548390 | website | automated encouragement | adul | cbt-based | cau | cut | some concerns | post | cbt | cau |
| Stuart, 2022 | 234 | 0.1815184 | 0.0210518 | 0.1450922 | website | automated encouragement | adul | cbt-based | cau | cut | some concerns | follow-up | cbt | cau |
| Sun, 2021 | 235 | 0.4809143 | 0.0353930 | 0.1881301 | mobile | minimal to no support | ppd | cbt-based | other ctr | cut | some concerns | post | 3rd | other ctr |
| Sun, 2021 | 236 | -0.1058622 | 0.0442927 | 0.2104583 | mobile | minimal to no support | ppd | cbt-based | other ctr | cut | some concerns | follow-up | 3rd | other ctr |
| Titov, 2010 | 237 | 1.0784313 | 0.0567081 | 0.2381346 | website | human encouragement | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Titov, 2010 | 238 | 1.0730864 | 0.0535555 | 0.2314207 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Titov, 2010 | 239 | 1.2584710 | 0.0593554 | 0.2436296 | website | human encouragement | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Titov, 2010 | 240 | 1.2617210 | 0.0561626 | 0.2369865 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Unlu-Ince, 2013 | 241 | 0.7091438 | 0.0764129 | 0.2764288 | website | guided | other group | not-cbt-based | wl | cut | some concerns | post | pst | wl |
| van Bastelaar, 2011 | 242 | 0.4915783 | 0.0255090 | 0.1597153 | website | guided | med | cbt-based | wl | cut | high risk | post | cbt | wl |
| Van Luenen, 2018 | 243 | 0.6089118 | 0.0222924 | 0.1493064 | website | guided | med | cbt-based | wl | cut | low risk | post | cbt | wl |
| Van Luenen, 2018 | 244 | 0.7635636 | 0.0228615 | 0.1512002 | website | guided | med | cbt-based | wl | cut | low risk | post | cbt | wl |
| Vernmark, 2010 | 245 | 0.9293583 | 0.0766157 | 0.2767954 | both | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Vernmark, 2010 | 246 | 0.8008440 | 0.0746462 | 0.2732146 | both | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Vernmark, 2010 | 247 | 0.5566486 | 0.0743652 | 0.2726999 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Vernmark, 2010 | 248 | 0.3559166 | 0.0726831 | 0.2695980 | website | guided | adul | cbt-based | wl | mdd | some concerns | post | cbt | wl |
| Vigod, 2021 | 249 | 0.1483359 | 0.0521728 | 0.2284137 | website | guided | ppd | not-cbt-based | wl | cut | high risk | post | ipt | wl |
| Warmerdam, 2008 | 250 | 0.5434037 | 0.0237089 | 0.1539771 | website | guided | adul | cbt-based | wl | cut | some concerns | post | cbt | wl |
| Warmerdam, 2008 | 251 | 0.4309754 | 0.0233932 | 0.1529484 | website | guided | adul | not-cbt-based | wl | cut | some concerns | post | pst | wl |
| Weise, 2019 | 252 | 0.9709331 | 0.0296132 | 0.1720847 | website | guided | adul | cbt-based | wl | mood | high risk | post | cbt | wl |
| Westerhof, 2019 | 253 | 0.2230608 | 0.1613205 | 0.4016473 | website | guided | adul | not-cbt-based | wl | cut | some concerns | post | lrt | wl |
| Westerhof, 2019 | 254 | 0.7078528 | 0.1859672 | 0.4312391 | website | guided | adul | not-cbt-based | wl | cut | some concerns | follow-up | lrt | wl |
| Williams, 2013a | 255 | 0.9394896 | 0.1063702 | 0.3261444 | website | guided | adul | cbt-based | wl | mdd | high risk | post | cbt | wl |
| Williams, 2013a | 256 | 0.9355157 | 0.1062781 | 0.3260031 | website | guided | adul | cbt-based | wl | mdd | high risk | post | cbt | wl |
| Williams, 2022 | 257 | 0.1274222 | 0.1116910 | 0.3342020 | website | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Ying, 2022 | 258 | 3.2995380 | 0.0432942 | 0.2080725 | website | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Ying, 2022 | 259 | 1.5625309 | 0.0238782 | 0.1545257 | website | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Ying, 2022 | 260 | 2.5786107 | 0.0335517 | 0.1831713 | website | guided | adul | cbt-based | wl | cut | high risk | post | cbt | wl |
| Zhao, 2022 | 261 | 1.2690885 | 0.0264824 | 0.1627342 | website | minimal to no support | young | cbt-based | wl | cut | some concerns | post | 3rd | wl |
Before proceeding with a multiverse meta-analysis, it’s important to ensure your data meets the requirements:
# Check if data meets requirements for multiverse analysis
check_data_multiverse(data_digDep)Data check passed. Dataset is valid.
[1] TRUE
The check_data_multiverse() function validates:
study, es_id, yi, vi, and wf_* columns)When conducting a multiverse meta-analysis, a critical step is determining which analytic choices (decision nodes) should be included in your multiverse. Not all potential choices merit inclusion, and an overly expansive multiverse can obscure meaningful effects with unjustified alternatives. Following Del Giudice and Gangestad (2021), we recommend a systematic approach to evaluating decision nodes.
Before including factors in your multiverse, assess them against these criteria:
For each decision node in your analysis, classify it into one of these three categories:
This approach leads to more interpretable and theoretically meaningful results by avoiding the creation of multiverse “black holes” where true effects are obscured by a sea of unjustified alternatives.
For our depression and inflammation example:
By applying this framework, we ensure our multiverse analysis is both comprehensive where appropriate and focused where theory provides clear guidance.
Next, we define the specifications to explore in our multiverse analysis:
# Define "Which factors" (analytical choices about data subsets)
wf_vars <- c("wf_1", "wf_2", "wf_3", "wf_4")
# Define meta-analytical methods
ma_methods <- c("reml", "fe", "3-level", "rve", "p-uniform")
# Define dependency handling approaches
dependencies <- c("aggregate", "modeled")
# Create specifications grid
specs <- create_multiverse_specifications(
data = data_digDep,
wf_vars = wf_vars,
ma_methods = ma_methods,
dependencies = dependencies
)
# Display number of specifications
tibble(
"Description" = "Number of valid specifications",
"Value" = specs$number_specs
) %>%
kable(format = "html") %>%
kable_styling(full_width = FALSE, position = "left")| Description | Value |
|---|---|
| Number of valid specifications | 1800 |
# Look at first few specifications
specs$specifications %>%
head(10) %>%
kable(format = "html") %>%
kable_styling(
bootstrap_options = c("striped", "hover", "condensed"),
full_width = FALSE
) %>%
column_spec(1:ncol(specs$specifications),
background = if_else(
grepl("total_", as.matrix(specs$specifications[1:10,])),
"#e8f4f8", "white"
))Warning in ensure_len_html(background, nrows, "background"): The number of
provided values in background does not equal to the number of rows.
| wf_1 | wf_2 | wf_3 | wf_4 | dependency | ma_method | row_id |
|---|---|---|---|---|---|---|
| website | minimal to no support | adul | not-cbt-based | aggregate | reml | 1 |
| mobile | minimal to no support | adul | not-cbt-based | aggregate | reml | 2 |
| both | minimal to no support | adul | not-cbt-based | aggregate | reml | 3 |
| total_wf_1 | minimal to no support | adul | not-cbt-based | aggregate | reml | 4 |
| website | guided | adul | not-cbt-based | aggregate | reml | 5 |
| mobile | guided | adul | not-cbt-based | aggregate | reml | 6 |
| both | guided | adul | not-cbt-based | aggregate | reml | 7 |
| total_wf_1 | guided | adul | not-cbt-based | aggregate | reml | 8 |
| website | automated encouragement | adul | not-cbt-based | aggregate | reml | 9 |
| mobile | automated encouragement | adul | not-cbt-based | aggregate | reml | 10 |
Each row in the specifications represents a unique analytical path through the multiverse. The columns indicate:
wf_* columns)dependency)ma_method)With our specifications defined, we can now run the multiverse analysis:
# Run the multiverse analysis
results <- run_multiverse_analysis(
data_multiverse = data_digDep,
specifications = specs$specifications,
how_methods = ma_methods
)
# View summary of results
summary_stats <- results %>%
summarize(
n_analyses = n(),
mean_es = mean(b, na.rm = TRUE),
median_es = median(b, na.rm = TRUE),
min_es = min(b, na.rm = TRUE),
max_es = max(b, na.rm = TRUE),
prop_significant = mean(pval < 0.05, na.rm = TRUE)
)
summary_stats %>%
tidyr::pivot_longer(cols = everything(), names_to = "Statistic", values_to = "Value") %>%
mutate(
Value = case_when(
Statistic == "prop_significant" ~ paste0(round(Value * 100, 1), "%"),
TRUE ~ round(Value, 3) %>% as.character()
),
Statistic = case_when(
Statistic == "n_analyses" ~ "Number of Analyses",
Statistic == "mean_es" ~ "Mean Effect Size",
Statistic == "median_es" ~ "Median Effect Size",
Statistic == "min_es" ~ "Minimum Effect Size",
Statistic == "max_es" ~ "Maximum Effect Size",
Statistic == "prop_significant" ~ "Proportion Significant (p < 0.05)",
TRUE ~ Statistic
)
) %>%
kable(format = "html", caption = "Summary of Multiverse Meta-Analysis Results") %>%
kable_styling(
bootstrap_options = c("striped", "hover"),
full_width = FALSE,
position = "center"
) %>%
row_spec(row = 0, bold = TRUE, color = "white", background = "#2c3e50")| Statistic | Value |
|---|---|
| Number of Analyses | 342 |
| Mean Effect Size | 0.48 |
| Median Effect Size | 0.468 |
| Minimum Effect Size | 0.028 |
| Maximum Effect Size | 2.791 |
| Proportion Significant (p < 0.05) | 91.2% |
The metaMultiverse package provides several visualization options to explore and interpret your results.
The specification curve visualizes the distribution of effect sizes across all specifications:
# Define a lookup table for factor labels
factor_label_lookup <- list(
wf_1 = "Technology Type",
wf_2 = "Guidance Level",
wf_3 = "Target Group",
wf_4 = "Therapy Type",
ma_method = "Meta-Analysis Method"
)
# Create the specification curve plot
plotly_descriptive_spec_curve(
data = results,
ylim_lower = -0.5,
ylim_upper = 1.5,
colorblind_friendly = TRUE,
factor_label_lookup = factor_label_lookup
)The specification curve consists of two panels:
The Vibration of Effects plot shows the relationship between effect sizes and p-values:
plotly_VoE(
data = results,
x = "b",
y = "pval",
colorblind_friendly = TRUE,
cutoff = 5, # Minimum number of studies
vertical_lines = c(0.1, 0.9), # Quantile reference lines
hline_value = 0.05 # Significance threshold
)When interpreting results from a multiverse meta-analysis, consider:
Consistency of direction: Do most specifications suggest effects in the same direction?
A finding is more robust if the majority of specifications point to effects in the same direction, even if magnitude varies.
Range of effect sizes: How much do effect sizes vary across specifications?
A narrow range suggests conclusions are stable across analytical choices, while wide ranges indicate high sensitivity to analytical decisions.
Analytical decisions driving variation: Which analytical choices have the largest impact on results?
Identify which factors (e.g., inclusion criteria, meta-analytical method) most strongly influence effect size estimates.
Proportion of significant results: What proportion of specifications yield statistically significant results?
A high proportion of significant results across the multiverse suggests a robust finding.
Decision sensitivity: Are conclusions robust to different analytical choices?
Assess whether key conclusions change substantively depending on analytical choices.
You can extend the package by adding custom meta-analytical methods:
# Example of defining a custom meta-analytical function
calculate_custom_method <- function(dat) {
# Your custom implementation here
# Return results in standardized format
list(
b = estimate,
ci.lb = lower_ci,
ci.ub = upper_ci,
pval = p_value
)
}
# Add to how_methods
custom_methods <- c(ma_methods, "custom_method")You can focus on a subset of specifications for targeted analysis:
# Focus on specifications with specific characteristics
subset_specs <- specs$specifications %>%
filter(
wf_1 == "specific_value",
dependency == "modeled"
)
# Run analysis on subset
#subset_results <- run_multiverse_analysis(
# data_multiverse = data_digDep,
# specifications = subset_specs,
# how_methods = ma_methods
#)The metaMultiverse package provides a systematic framework for exploring the impact of analytical decisions in meta-analysis. By conducting and visualizing multiple analyses simultaneously, researchers can:
For more information, see the package documentation or contact the developers.